It’s an amazing act of trust in the world to not aim to be a person that moves the world directly.
The most extreme case is being a monk. A monk relies on other people to provide for him. He does create value to the world, because he provides people with peace and embodies a mindset that others can aspire to. However, there is no reason that people have to give alms to a monk, even though this is true. Being a scholar is actually similar. Even though what a scholar does isn’t directly useful, ey do it anyway, and trust society to allow em to do what they do.
The opposite kind of person is the one who seeks power. This can be for good means or bad. A good person who seeks power seeks it because ey know that with capital—money, a leadership position, etc.—they can mobilize the world to carry out their aims. They may prioritize getting power over developing their moral or reflective self, and that’s fine.
However, I fear the kind of person who, in this pursuit, becomes narrowminded. They think that in order to change the world the first step is to gain capital, and discount anyone who doesn’t think a similar way, even if they are good, talented, and/or hardworking people.
There is something that a president and a monk can learn from one another. And even though the president is more conventionally successful, that doesn’t mean they’re more valuable. (One can argue this, but that’s not the point.) The currencies are fundamentally different. Each has something the other lacks. The president deals with currencies in terms of money, support of other countries and politicians, ability to start initiatives and enact laws, etc.—conventional power. The monk’s currency is the mind that ey has cultivated, compassion, clear thought, happiness. (There are religious scholars who have had profound effect—Thich Nhat Hahn, Karen Armstrong, etc.) For a monk to enact wide change they need the support of other people, and they don’t have the power that a president can easily draw on. They make appeals, show by example what a good life is and hope that others will find it helpful enough to spread. (Perhaps the monk convinces the president that there is a better way than conflict, and the president listens. The president uses the power at his disposal to prevent the conflict. If you count by capital, the president did 100% of the work in stopping the conflict. But is that the right credit assignment? No.) A teacher is the same way—a teacher’s power doesn’t by default reach past the classroom—but a teacher can inspire a student who later obtains more resources to do a big thing.
So being a monk or a teacher instead of a businessman is in some sense, having a kind of faith in society. So is being a writer, in a way. Suppose you want to stop war. You can go the power route—become the president, and be in a position to veto any war. Or you can do something different. You can write a fiction book where you portray war honestly, and you foment some anti-war thoughts in your readers. You have so little direct control over what happens. It’s impossible to measure how much your book led to a decrease of conflict in the world. But that doesn’t mean that its effect is nonexistent.
There’s a state of mind I get in when I’m around people who I conceive as going after power, and I hate it. I feel like I have to be like them. There are certain people who want to be at the helm, and that’s great—but it’s not good when those people disdain the others for “lack of ambition.”
I don’t donate to the homeless person on the street in order to be effective, and neither do I donate because I want to buy a warm glow. I do it to remind myself that I am more fortunate than other people, that I should be more willing to give, to remind myself that there is a world of suffering and that the action of giving a dollar is inadequate. I do it as a “knight of infinite resignation.” I do it so I can remember to keep giving. After giving I feel more inadequate than if I didn’t give, and that is a good feeling.
This kind of fear is what makes me averse to a callous brand of transhumanism and superintelligence. I’m not averse to superintelligent machines. I’m concerned that goodness and morality will be wiped from the picture. Before we get to superintelligence, we’ll have weak AI in the sense of intelligence augmentation and AI’s that take a skilled programmer to train/control.
At this stage there will be a larger gap than ever before between the poor and the rich, because there’s a new kind of capital here—computing resources, computer programs, and programming skills as it relates to AI. The people who have these resources become vastly more powerful than those with just power or control over people, because they have control over AI. How can we be sure that these people will not abuse their AI power? While I’m saying “AI” here, I don’t necessarily mean strong AI. I just mean powerful computer programs (which we have already). The people who have computing power get to set the goals of the AI—and they can set the goal to be making money for the company. Even if they aren’t so explicit, their incentive is not completely aligned to the welfare of the world.
We will have to “teach” AI. And I think people who are compassionate but not technically adept will have a lot to teach. Or people who study humanities. I don’t mean this in the sense that we’re going to pass them the laptop and have them write a “compute_morality” function, I mean that we should take inspiration from them. We shouldn’t write them out of the picture. What I fear from transhumanists is that they’re so desperate for superintelligence that they disdain all the intelligence already around them as unworthy—especially if it’s this intelligence is not math/science but intelligence in the sense of wisdom, knowing what’s good. They’re so desperate for superintelligence that they already mark as obsolete the people around them, or even want those people to mark themselves as obsolete.
The monk doesn’t tell the president what policies to enact, what the terms of the treaties are. But the monk, in eir interaction with the president, communicates a set of values which the president internalizes, even if just briefly, in eir consideration of the policies and treaties to draft.
This may just be a romanticization, wanting everyone to have some value that can’t be trumped. But I think that this is of practical significance, because programming a set of values into a machine is hard, and while math and computer science and logic is necessary, it isn’t sufficient, because we need people who embody those values and study those values, and that’s often different from the math/CS people.
What is it about compassion that can’t be expressed by a utility function? In the end it could just be a utility function. And maybe if we have the explicit program that computes an approximation, it’s important. But the programmers will come up with a simple utility function that turns out to be too simple.
The reason why System 1 feelings like sympathy, love, etc. are useful is that it’s very hard to write a function that says for example is something good?—for every rule there’s probably a counterexample. If we had to go through such a mental checklist every time to find the appropriate response, we wouldn’t do very well. People study ethics because this is a hard problem. In order for us to come up with the right utility function, we need these people.
I’m terrified not because we don’t have the science of AI safety worked out—but because theory of AI safety or no, bad humans can abuse power, and AI has the ability to do worse things than nuclear weapons, and the ability to do things more by accident (for nukes, you’d have to at least consciously push a button). The more tech we have, the easier it is for bad humans to do more bad things, which is why good vs. evil never stops. (Some people use this as a reason against progress.)
I fear we’re heading towards a technocracy. I’m worried because education is not up to par. We aren’t even teaching kids high school math well. I don’t want 90% of society to be made up of sheeple whose main job is to consume, and the other 10% to be building the machines and giving them their value functions without consulting the other 90%. I want a world where everyone can do a little programming of smart systems, rather than bang their heads when the Internet of Things around them is malfunctioning. It’s a world that can come to pass when we significantly raise the level of programming literacy.